{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PKrEJX4IuTN3"
   },
   "source": [
    "# Adding a dense layer to VGG\n",
    "\n",
    "The last layer of Vgg16 outputs a vector of 1000 categories, because that is the number of categories the competition asked for. Of these categories, some of them certainly correspond to cats and dogs, but at a much more granular level (specific breeds).\n",
    "\n",
    "We will simply add a Dense layer on top of the imagenet layer, and train the model to map the imagenet classifications of input images of cats and dogs to cat and dog labels.\n",
    "\n",
    "Note that this is not what we have been doing in the very first lecture!\n",
    "\n",
    "Have a look at [CS231n: Linear Classification](http://cs231n.github.io/linear-classify/) for more precisions and especially to [CS231n: Softmax classifier](http://cs231n.github.io/linear-classify/#softmax) if you had trouble with logistic regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KyxtNaA0uTN5"
   },
   "source": [
    "## 1. Preparations"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "y-_LIRVGuhqC"
   },
   "source": [
    "# to install pytorch on colab\n",
    "from os import path\n",
    "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
    "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
    "\n",
    "accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'\n",
    "\n",
    "!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-{platform}-linux_x86_64.whl torchvision\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "acOo9KHRugOi"
   },
   "source": [
    "!pip install -U bcolz"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "K79BtvaUu1Bw"
   },
   "source": [
    "!pip install Pillow==4.0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "PG0R8S_auTN5"
   },
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "from torchvision import models,transforms,datasets\n",
    "import bcolz\n",
    "import time\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zbiLBOHcuTOA"
   },
   "source": [
    "We did precompute the outputs of Vgg16 model on our dataset (with Colab) and stored these values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Lnsjg6XguTOB"
   },
   "outputs": [],
   "source": [
    "use_gpu = torch.cuda.is_available()\n",
    "print('Using gpu: %s ' % use_gpu)\n",
    "\n",
    "dtype = torch.FloatTensor\n",
    "if use_gpu:\n",
    "    dtype = torch.cuda.FloatTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "6lp57KjUuTN9"
   },
   "outputs": [],
   "source": [
    "def load_array(fname):\n",
    "    return bcolz.open(fname)[:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "UnSiC7vPuTOG"
   },
   "outputs": [],
   "source": [
    "#where you stored your features\n",
    "data_dir_colab = '/home/lelarge/courses/data/colab/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "Sq09hBt6uTO_"
   },
   "outputs": [],
   "source": [
    "feat_train = load_array(os.path.join(data_dir_colab,'vgg16','feat_train.bc'))\n",
    "lbs_train = load_array(os.path.join(data_dir_colab,'vgg16','lbs_train.bc'))\n",
    "feat_val = load_array(os.path.join(data_dir_colab,'vgg16','feat_val.bc'))\n",
    "lbs_val = load_array(os.path.join(data_dir_colab,'vgg16','lbs_val.bc'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uGbUxhpJuTPC"
   },
   "source": [
    "## 2. Linear model for VGG16 features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kmfZSTJluTPC"
   },
   "source": [
    "We are now ready to define our linear model.\n",
    "\n",
    "For more details about the [cross entropy cost function](http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "s0V9GnE2uTPC"
   },
   "outputs": [],
   "source": [
    "lm = torch.nn.Sequential(\n",
    "    torch.nn.Linear(1000, 2),\n",
    "    torch.nn.LogSoftmax(dim = 1)\n",
    ")\n",
    "loss_fn = torch.nn.NLLLoss(size_average=False)\n",
    "if use_gpu:\n",
    "    lm = lm.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wo1UF3fKuTPF"
   },
   "source": [
    "Since our features are currently stacked in a _numpy ndarray_, we need to create a dataset of tensors and then a dataloader.\n",
    "\n",
    "For the dataset, you can use _torch.from_numpy_, _torch.tensor_ and _zip_\n",
    "\n",
    "For the dataloader, you should use _torch.utils.data.DataLoader_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "nnjzGy9JuTPF"
   },
   "outputs": [],
   "source": [
    "bs = 128\n",
    "\n",
    "train_dataset = # your code\n",
    "test_dataset = # your code\n",
    "train_loader = torch.utils.data.DataLoader(#your code here)\n",
    "test_loader = torch.utils.data.DataLoader(#your code here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5psdVrkfuTPJ"
   },
   "source": [
    "### 2.1 Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QJl3OZVluTPJ"
   },
   "source": [
    "We define next a holistic training function (```train_model```) that will:\n",
    "- run for a pre-defined number of epochs/iterations\n",
    "- fetch training samples randomly during each epoch(all samples are used during an epoch)\n",
    "- pass samples through network, compute error, gradients and updates network parameters\n",
    "- keep and print training statistics: training loss, accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "WqsjAHC5uTPJ"
   },
   "outputs": [],
   "source": [
    "def train_model(model,size,data_loader=None,epochs=1,optimizer=None):\n",
    "    model.train()\n",
    "    loss_t = np.zeros(epochs)\n",
    "    acc_t = np.zeros(epochs)\n",
    "    for epoch in range(epochs):\n",
    "        \n",
    "        running_loss = 0.0\n",
    "        running_corrects = 0\n",
    "        for inputs,classes in data_loader:\n",
    "                            \n",
    "            #\n",
    "            # your code\n",
    "            #\n",
    "            running_loss += # your code\n",
    "            running_corrects += # your code\n",
    "            \n",
    "        epoch_loss = running_loss / size\n",
    "        epoch_acc = running_corrects.data.item() / size\n",
    "        \n",
    "        loss_t[epoch] = epoch_loss\n",
    "        acc_t[epoch] = epoch_acc\n",
    "    print('Loss: {:.4f} Acc: {:.4f}'.format(epoch_loss, epoch_acc))\n",
    "    return loss_t, acc_t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sx_C-tbMuTPM"
   },
   "source": [
    "We set our hyperparameters:\n",
    "- learning rate\n",
    "- optimizer to be used for gradient descent, here SGD (Stochastic Gradient Descent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "AYnppJ8DuTPN"
   },
   "outputs": [],
   "source": [
    "learning_rate = 1e-4\n",
    "optimizer_lm = torch.optim.SGD(lm.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dset_sizes = {'train': 23000, 'valid': 2000}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hJpimXuLuTPR"
   },
   "source": [
    "We train our model for 100 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vFsn7n0yuTPR"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "loss1, acc1 = train_model(model=lm,size=dset_sizes['train'],data_loader = train_loader ,epochs=100,optimizer=optimizer_lm)\n",
    "#loss1, acc1 = (train_model(model=lm,size=dset_sizes['train'],feat=feat_train,labels=lbs_train, epochs=100,optimizer=optimizer_lm,batch_size = 64,shuffle=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r-c_SKJHuTPW"
   },
   "source": [
    "We plot the evolution of the training loss across epochs. \n",
    "\n",
    "Ideally is should have a steep descent in the first epochs, then decrease smoothly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4gBydZyyuTPY"
   },
   "outputs": [],
   "source": [
    "plt.plot(loss1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JlA0t6bnuTPc"
   },
   "source": [
    "We plot the evolution of the accuracy of our model on the training data. The behavior resembles globally to the one of the loss: big improvement at the beginning, then smaller improvements as training advances.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZQCfalL8uTPd"
   },
   "outputs": [],
   "source": [
    "plt.plot(acc1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8wbmWXMnuTPf"
   },
   "source": [
    "The __loss__ helps the network to learn and update the parameters according to the criterion that we give to the network.\n",
    "\n",
    "The __accuracy__ on the other hand is a performance metric for the task for which we want to use the network for. In many cases the accuracy cannot be integrated as a loss/criterion function, so we need to identify or design loss functions that will guide the model towards the behavior we wish to have for our task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LUhPRq5tuTPf"
   },
   "source": [
    "Next we let the model train for 100 additional epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LCx-Et3NuTPh"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "loss2, acc2 = train_model(model=lm,size=dset_sizes['train'],data_loader =train_loader ,epochs=100,optimizer=optimizer_lm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KAWCyfpouTPj"
   },
   "source": [
    "Again we plot the loss and accuracy for the current training interval: _epochs[100:200]_.\n",
    "What changes do you notice? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jnll44NguTPj"
   },
   "outputs": [],
   "source": [
    "plt.plot(loss2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TRPiQRqxuTPm"
   },
   "outputs": [],
   "source": [
    "plt.plot(acc2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MQdbyFBSuTPr"
   },
   "source": [
    "We train the model train for 100 more epochs and plot the evolution of our training indicators. \n",
    "How are they evolving comparing to the previous runs?\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "acdwL1zauTPs"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "loss3, acc3 = train_model(model=lm,size=dset_sizes['train'],data_loader =train_loader ,epochs=100,optimizer=optimizer_lm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0Rc5WuaPuTPw"
   },
   "outputs": [],
   "source": [
    "plt.plot(loss3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jfEuKrkmuTP2"
   },
   "outputs": [],
   "source": [
    "plt.plot(acc3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S7CaET8muTP7"
   },
   "source": [
    "### 2.2 Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "v452HrVIuTP8"
   },
   "source": [
    "We define next a holistic test function (```test_model```) that will:\n",
    "- fetch test samples\n",
    "- pass samples through network, compute error, accuracy and predictions\n",
    "- keep and print test statistics: test loss, accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "oaENq9k1uTP9"
   },
   "outputs": [],
   "source": [
    "def test_model(model,size,data_loader=None):\n",
    "    model.eval()\n",
    "    \n",
    "    predictions = np.zeros(size)\n",
    "    running_loss = 0.0\n",
    "    running_corrects = 0\n",
    "    count = 0 \n",
    "    for inputs,classes in data_loader:\n",
    "        #\n",
    "        # your code\n",
    "        #\n",
    "        count +=1\n",
    "        \n",
    "    print('Loss: {:.4f} Acc: {:.4f}'.format(running_loss / size, running_corrects.data.item() / size))\n",
    "    return predictions, running_loss / size, running_corrects.data.item() / size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y02ql1rHuTQA"
   },
   "source": [
    "We evaluate on the test data a snapshot of our model at _epoch #300_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BIRhbtPFuTQC"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "preds, loss_val, acc_val = test_model(model=lm,size=dset_sizes['valid'],data_loader=test_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9xYPDhdNuTQG"
   },
   "outputs": [],
   "source": [
    "loss_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S-4W062ouTQI"
   },
   "source": [
    "## 3. Quantitative analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sy4ZTZHWuTQK"
   },
   "source": [
    "We concatenate the training losses across the 300 training epochs and plot them along with the loss on the test data using a snapshot of our model at epoch #300.\n",
    "\n",
    "What do you notice? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QtykLETguTQK"
   },
   "outputs": [],
   "source": [
    "plt.plot(np.concatenate((loss1, loss2, loss3)))\n",
    "plt.plot([loss_val]*300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KMKOQuOvuTQP"
   },
   "source": [
    "We illustrate a similar plot for the training loss values at _epochs[200:300]_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wofAeNVQuTQP"
   },
   "outputs": [],
   "source": [
    "plt.plot(loss3)\n",
    "plt.plot([loss_val]*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jHQ95XxhuTQW"
   },
   "source": [
    "We now illustrate the aggregated training accuracies on epochs[0:300] along with the test accuracy for the model at epoch #300."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wzwKaVAPuTQX"
   },
   "outputs": [],
   "source": [
    "plt.plot(np.concatenate((acc1, acc2, acc3)))\n",
    "plt.plot([acc_val]*300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MDHMaNPquTQZ"
   },
   "source": [
    "We train our model for 1000 more epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OfqfeVsuuTQc"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "loss4, acc4 = train_model(model=lm,size=dset_sizes['train'],data_loader =train_loader ,epochs=1000,optimizer=optimizer_lm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8g4NOeCPuTQe"
   },
   "source": [
    "We test the model snapshot at _epoch #1300_ and keep its statiscs and performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "6kGzqTgcuTQf"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "preds2, conf2, loss_val2, acc_val2 = (test_model(model=lm,size=dset_sizes['valid'],feat=feat_val,labels=lbs_val,batch_size=2000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ma5_9t_BuTQh"
   },
   "source": [
    "We aggregate train loss values at _epochs[300:1300]_ and test loss at _epochs[300]_ and _epochs[1300]_.\n",
    "Do you notice a trend?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "njq90A-7uTQh"
   },
   "outputs": [],
   "source": [
    "plt.plot(np.concatenate((loss3,loss4)))\n",
    "plt.plot(np.concatenate(([loss_val]*100,[loss_val2]*1000)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "biVd0Eicb66Y"
   },
   "source": [
    "A similar plot for the accuracy values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "fuAt-2gruTQm"
   },
   "outputs": [],
   "source": [
    "plt.plot(np.concatenate((acc1, acc2, acc3, acc4)))\n",
    "plt.plot(np.concatenate(([acc_val]*300,[acc_val2]*1000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "acc_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HMZBihNguTQr"
   },
   "source": [
    "## Exercise\n",
    "\n",
    "What is happening? \n",
    "\n",
    "Make better plots on which we see the evolution of the loss/accuracy on both the training and validation sets as a function of the number of epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gdDueKxUuTQs"
   },
   "source": [
    "## 4. Viewing model prediction (qualitative analysis)\n",
    "\n",
    "The most important metrics for us to look at are for the validation set, since we want to check for over-fitting.\n",
    "\n",
    "With our first model we should try to overfit before we start worrying about how to handle that - there's no point even thinking about regularization, data augmentation, etc if you're still under-fitting! (We'll be looking at these techniques after the 2 weeks break...)\n",
    "\n",
    "\n",
    "As well as looking at the overall metrics, it's also a good idea to look at examples of each of:\n",
    "\n",
    "   1. A few correct labels at random\n",
    "   2. A few incorrect labels at random\n",
    "   3. The most correct labels of each class (ie those with highest probability that are correct)\n",
    "   4. The most incorrect labels of each class (ie those with highest probability that are incorrect)\n",
    "   5. The most uncertain labels (ie those with probability closest to 0.5).\n",
    "\n",
    "In general, these are particularly useful for debugging problems in the model. Since our model is very simple, there may not be too much to learn at this stage..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "2zlK4iJ2uTQs"
   },
   "outputs": [],
   "source": [
    "# Number of images to view for each visualization task\n",
    "n_view = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BGmBImB7uTQv"
   },
   "source": [
    "Selecting correct predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "i7PM-ENHuTQx"
   },
   "outputs": [],
   "source": [
    "correct = np.where(preds==lbs_val)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "BeafrtLZuTQz"
   },
   "outputs": [],
   "source": [
    "from numpy.random import random, permutation\n",
    "idx = permutation(correct)[:n_view]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "csd3SW2FuTQ2"
   },
   "outputs": [],
   "source": [
    "idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "J-vni-AQuTQ5"
   },
   "outputs": [],
   "source": [
    "def imshow(inp, title=None):\n",
    "#   Imshow for Tensor.\n",
    "    inp = inp.numpy().transpose((1, 2, 0))\n",
    "    mean = np.array([0.485, 0.456, 0.406])\n",
    "    std = np.array([0.229, 0.224, 0.225])\n",
    "    inp = std * inp + mean\n",
    "    plt.imshow(inp)\n",
    "    if title is not None:\n",
    "        plt.title(title)\n",
    "    plt.pause(0.001)  # pause a bit so that plots are updated\n",
    "\n",
    "data_dir = '/home/lelarge/courses/data/dogscats'\n",
    "normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
    "\n",
    "prep1 = transforms.Compose([\n",
    "                transforms.CenterCrop(224),\n",
    "                transforms.ToTensor(),\n",
    "                normalize,\n",
    "            ])\n",
    "dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), prep1)\n",
    "         for x in ['train', 'valid']}\n",
    "\n",
    "dataset_correct = torch.utils.data.DataLoader([dsets['valid'][x] for x in idx],batch_size = n_view,shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "GVQ9iM5buTQ8"
   },
   "outputs": [],
   "source": [
    "for data in dataset_correct:\n",
    "    inputs_cor,labels_cor = data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "7mdvWiSPuTQ9"
   },
   "outputs": [],
   "source": [
    "# Make a grid from batch\n",
    "out = torchvision.utils.make_grid(inputs_cor)\n",
    "\n",
    "imshow(out, title=[x for x in labels_cor])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "tefNitxUuTRA"
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image, display\n",
    "for x in idx:\n",
    "    display(Image(filename=dsets['valid'].imgs[x][0], retina=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a-iCNKZSuTRB"
   },
   "source": [
    "Selecting incorrect predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "VxmVjAI1uTRC"
   },
   "outputs": [],
   "source": [
    "incorrect = np.where(preds!=lbs_val)[0]\n",
    "for x in permutation(incorrect)[:n_view]:\n",
    "    print(dsets['valid'].imgs[x][1])\n",
    "    display(Image(filename=dsets['valid'].imgs[x][0], retina=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "cJX3_d_2uTRF"
   },
   "outputs": [],
   "source": [
    "#3. The images we most confident were cats, and are actually cats\n",
    "correct_cats = np.where((preds==0) & (preds==lbs_val))[0]\n",
    "most_correct_cats = np.argsort(conf[correct_cats,1])[:n_view]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "1rcEbuNUuTRH"
   },
   "outputs": [],
   "source": [
    "for x in most_correct_cats:\n",
    "    display(Image(filename=dsets['valid'].imgs[correct_cats[x]][0], retina=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "8Y8RCsajuTRL"
   },
   "outputs": [],
   "source": [
    "#3. The images we most confident were dogs, and are actually dogs\n",
    "correct_dogs = np.where((preds==1) & (preds==lbs_val))[0]\n",
    "most_correct_dogs = np.argsort(conf[correct_dogs,0])[:n_view]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "FgcCh_deuTRN"
   },
   "outputs": [],
   "source": [
    "for x in most_correct_dogs:\n",
    "    display(Image(filename=dsets['valid'].imgs[correct_dogs[x]][0], retina=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4ShVf7O_uTRP"
   },
   "source": [
    "## Exercise\n",
    "\n",
    "As seen in the first lecture, the last layer of Vgg16 is simply a dense layer that outputs 1000 elements. Therefore, it seems somewhat unreasonable to stack a dense layer meant to find cats and dogs on top of one that's meant to find imagenet categories, in that we're limiting the information available to us by first coercing the neural network to classify to imagenet before cats and dogs...\n",
    "\n",
    "Instead, do finetuning, i.e remove that last layer and add on a new layer for cats and dogs. \n",
    "\n",
    "Compare to what we did in the first lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "FRdJVGg_uTRQ"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "HMZBihNguTQr"
   ],
   "name": "04_dogscast_colab.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python [conda env:pytorch]",
   "language": "python",
   "name": "conda-env-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
